I've tested this on Linux and Windows 7, both serial console and uploads to a Arduino Micro work. It should also be tested on Windows 10 and OSX, both of which use the USB descriptor to determine what driver to load (so we should confirm that they still load the right driver).

If this turns out to work reliably, that the same change should be made for 16u2 firmware for AVR here and for SAM here (I did not test or confirm those changes yet). For the SAMD core, it seems this change was already made in commit arduino/ArduinoCore-samd@5873b03. There might be other places too.

@cmaglie, @facchinm, would be great if you could have a look at this, since I believe making these changes will greatly help to prevent issues with ModemManager (at least for the the native USB devices). Since there are a lot of devices out there with fixed (and broken) 16u2 firmwares, I'll make sure to report a bug at Debian to consider re-enabling the blacklist as well (and probably at upstream ModemManager to consider keeping the blacklist around as well).

https://gitlab.freedesktop.org/mobile-broadband/ModemManager/issues/111

Just in case it serves: removing modemmanager - as seen in the aboved mentioned issue #7690 - solved the same problem with Arduino IDE 1.8.9 on Ubuntu 18.04.2 (with an Arduino Leonardo R3 board).

Can someone let me know which is the full list of USB vid:pids that expose ttyACM ports that may advertise bFunctionProtocol=1? I'm trying to compile a short list of devices to blacklist in ModemManager while using the strict filter.

Otherwise, is there any other kind of check we could do in ModemManager to automatically detect that this is an Arduino device wrongly claiming AT protocol support? E.g. I've seen a full lsusb output of one of these devices (vid 0x2341, pid 0x8036), that also exposes a HID USB interface in addition to the ttyACM port. Is that a general rule in this kind of devices? i.e. do they always expose a ttyACM port and a HID interface?

@aleksander0m It would be great if you could blacklist any board with VID 0x2341 (regardless of the PID) so that they are not probed by ModemManager on enumeration. Alternatively I could provide you with a VID/PID list (where VID is always 0x2341). This has the disadvantage of increasing the maintainance burden because ModemManager's blacklist needs to be updated with a new VID/PID pair every time a new board is released.

i.e. do they always expose a ttyACM port and a HID interface?

Nope, I believe this is only when the HID libraries are loaded. By default, it's just TTY.

It would be great if you could blacklist any board with VID 0x2341

This catches Arduino devices, but there are some other suppliers as well (such as Sparkfun, Adafruit, Teensy, Seeed and probably a dozen others that might have devices with native USB). There's also a lot of third-party clones, but they often use the main Arduino core and bootloader, which will probably use 0x2341 for both.

Also note that, apart from fixing the core, I think the bootloader also needs to be fixed (which will only affect newly produced boards, not existing ones).

Also see https://gitlab.freedesktop.org/mobile-broadband/ModemManager/issues/127 for upstream discussion.

Thanks for the comments. The problem seems much more broad than I though, if multiple suppliers with different VIDs are affected, so we should probably remove the "AT-capable ttyACM port" heuristic from ModemManager all together, as we cannot trust that info :/

@aleksander0m - Maybe also reconsider whether to disregard udev rules with ID_MM_DEVICE_IGNORE with strict filter policy? Many Linux distros are now making "strict" their default, probably under the assumption they're doing end users a favor by making their systems more secure. But really, what benefit could disregarding an explicit user choice to tell Modem Manager not to mess with a particular device possibly have?

@PaulStoffregen that is being taken care of, see https://gitlab.freedesktop.org/mobile-broadband/ModemManager/merge_requests/138
I will probably have that logic added in the next stable MM 1.10 update, along with the logic to disable the "AT-capable ttyACM port" heuristic.

@aleksander0m - I looked at that PR briefly. I still see "This tag is ignored when the STRICT filter policy is used" in the comments before the new ID_MM_TTY_BLACKLIST define. Why oh why would ModemManager ignore this explicit user choice with that filter mode? Many Linux distros are now using strict filter policy as their default setting.

@PaulStoffregen the ID_MM_DEVICE_IGNORE tag was ignored in strict filter because that is the tag used in the built-in blacklist, and in strict filter mode we fully ignore the blacklists. The MR I'm suggesting will keep "ID_MM_DEVICE_IGNORE" as a tag that users can use to ignore devices, and it would now apply in all filter modes, including strict. This MR tries to solve that specific issue you're experiencing, by making ID_MM_DEVICE_IGNORE really a user-usable tag (not just an internally used one).

There's a bit of history behind the filter modes, but the idea is to get all blacklists removed at some point in a future release, which is why we implemented the strict filter mode that uses heuristics to try to "guess" whether a device is a modem or not without using blacklists. What I didn't expect is devices wrongly claiming AT protocol support in the ACM interface, as in this Arduino bug... for this specific issue with the Arduino USB interface, this is the change I'm suggesting in ModemManager: https://gitlab.freedesktop.org/mobile-broadband/ModemManager/merge_requests/143

BTW, the new ID_MM_TTY_BLACKLIST in the MR would be an "internal-use-only" tag, users shouldn't use it, which is why we would ignore it in strict filter mode.

I'm really glad to hear this will be fixed. I hope you can understand what a terrible tech support nightmare the combination distros defaulting to strict filter policy and ModemManager's design to disregard udev-based ID_MM_DEVICE_IGNORE has inflicted on so many using Linux.

@PaulStoffregen As you can imagine, none of the changes done in ModemManager were done to make users unhappy; it's actually the opposite. The new "strict" mode was hopefully enough to finally be able to get rid of the blacklists we were maintaining, but sadly one of the heuristics added in the new mode was failing due to the wrongly claimed AT support in the Arduinos. If this has caused a "terrible tech support nightmare", well, I'm truly sorry, that was not the intention. I'm sure that the intention of the ttyACM port in the Arduino claiming AT support was also not on purpose or trying to make users unhappy.

Given that I cannot fix all arduinos out there, I'm trying to minimize the problem in ModemManager by explicitly ignoring those that claim protocol=1 in the ACM interface. And also, I'm also making ID_MM_DEVICE_IGNORE a "user usable tag" (it wasn't before) so that it also applies in strict mode. But one thing is clear: strict mode will still ignore all blacklists we ship (the blacklist will use a different tag name, thats all)

I do understand that this may have caused a "terrible tech support nightmare", there's no need for you to hope it. Would you be able to test the MM branches I'm suggesting to make sure they solve your issues?

Removing or updating the "AT-capable ttyACM port" heuristic ends up breaking device detection for several known modems, so we cannot workaround this issue in ModemManager just with that change, we need to improve heuristics.

Can anyone let me know whether Arduino devices affected by this issue will expose always a single ttyACM port, or could it be that depending on the build the device exposes >1 ttyACM port? @matthijskooijman

All devices I am aware of expose only a single ttyACM port, certainly in the bootloader.

I think it is possible for an application written for Arduino by a user to expose multiple ports, but I would suspect this is an unlikely case, and even if it does happen, such a user should be able to update their code to turn off the AT-capable bit and prevent this issue. So ignoring devices with a single ttyACM port (which I think is your suggestion?) would seem like an effective solution.

New approach suggested in MM: assuming that modems exposing ttyACM ports always expose more than one: https://gitlab.freedesktop.org/mobile-broadband/ModemManager/merge_requests/152

I know LUFA offers "Dual Virtual Serial Device" which would be two ttyACM ports.

http://www.fourwalledcubicle.com/LUFA.php

@PaulStoffregen does that device expose 2 ttyACM ports with class=2/subclass=2/protocol=1 (i.e. both ports reporting themselves as AT-capable)? That is the only case that would affect the new MM rules

@aleksander0m Thank you for working with us on this. I do not know what type of ttyACM ports the LUFA framework might expose but I'm quite confident that exposing 2 ttyACM ports which are both reporting themselves to be AT-capable is a negligible edge case. It's considerable more important to exclude all those Arduino boards which are exposing only 1 ttyACM port (I've also observed MM trying to talk to a Arduino MKR 1000 - so the issue is affecting more cores than just ArduinoCore-avr).

The changes we've been working on in upstream ModemManager are now released in MM 1.10.6, hopefully solving all these headaches https://lists.freedesktop.org/archives/modemmanager-devel/2019-September/007424.html

Even if this should no longer affect the MM logic, I would definitely suggest anyway to update arduino in order to avoid reporting itself as AT-capable in the ttyACM USB interface settings...